Autonomous Deep Learning: Continual Learning Approach for Dynamic Environments

The feasibility of deep neural networks (DNNs) to address data stream problems still requires intensive study because of the static and offline nature of conventional deep learning approaches. A deep continual learning algorithm, namely autonomous deep learning (ADL), is proposed in this paper. Unlike traditional deep learning methods, ADL features a flexible structure where its network structure can be constructed from scratch with the absence of an initial network structure via the self-constructing network structure. ADL specifically addresses catastrophic forgetting by having a different-depth structure which is capable of achieving a trade-off between plasticity and stability. Network significance (NS) formula is proposed to drive the hidden nodes growing and pruning mechanism. Drift detection scenario (DDS) is put forward to signal distributional changes in data streams which induce the creation of a new hidden layer. The maximum information compression index (MICI) method plays an important role as a complexity reduction module eliminating redundant layers. The efficacy of ADL is numerically validated under the prequential test-then-train procedure in lifelong environments using nine popular data stream problems. The numerical results demonstrate that ADL consistently outperforms recent continual learning methods while characterizing the automatic construction of network structures

DEVDAN: Deep Evolving Denoising Autoencoder

The Denoising Autoencoder (DAE) enhances the flexibility of the data stream method in exploiting unlabeled samples. Nonetheless, the feasibility of DAE for data stream analytic deserves an in-depth study because it characterizes a fixed network capacity that cannot adapt to rapidly changing environments. Deep evolving denoising autoencoder (DEVDAN), is proposed in this paper. It features an open structure in the generative phase and the discriminative phase where the hidden units can be automatically added and discarded on the fly. The generative phase refines the predictive performance of the discriminative model exploiting unlabeled data. Furthermore, DEVDAN is free of the problem-specific threshold and works fully in the single-pass learning fashion. We show that DEVDAN can find competitive network architecture compared with state-of-the-art methods on the classification task using ten prominent datasets simulated under the prequential test-then-train protocol.Comment: This paper has been accepted for publication in Neurocomputing 2019. arXiv admin note: substantial text overlap with arXiv:1809.0908

Low-Frequency Oscillation Mitigation usin an Optimal Coordination of CES and PSS based on BA

Small signal stability represents the reliability of generator for transferring electrical energy to the consumers. The stress of the generator increases proportionally with the increasing number of load demand as well as the uncertainty characteristic of the load demand. This condition makes the small signal stability performance of power system become vulnerable. This problem can be handled using power system stabilizer (PSS) which is installed in the excitation system. However, PSS alone is not enough to deal with the uncertainty of load issue because PSS supplies only an additional signal without providing extra active power to the grid. Hence, utilizing capacitor energy storage (CES) may solve the load demand and uncertainty issues. This paper proposes a coordination between CES and PSS to mitigate oscillatory behavior of the power system. Moreover, bat algorithm is used as an optimization method for designing the coordinated controller between CES and PSS. In order to assess the proposed method, a multi-machine two-area power system is applied as the test system. Eigenvalue, damping ratio, and time domain simulations are performed to examine the significant results of the proposed method. From the simulation, it is found that the present proposal is able to mitigate the oscillatory behavior of the power system by increasing damping performance from 4.9% to 59.9%

Autonomous deep learning for continual learning in complex data stream environment

The last decade has seen growing attention to the processing of infinite data sequences that are quickly generated in an online fashion. These data can be in the form of structured data or unstructured data. In this situation, a continual learning algorithm is required to continually learn and craft important knowledge from the data, hence offering positive forward transfer It refers to the impact that learning a task has on future task performance gem2017paz. A model having a high positive forward transfer is good at utilizing the past knowledge to improve its prediction in the incoming task., as well as being able to memorize previously seen tasks. That is, successfully circumventing the catastrophic forgetting problem The catastrophic forgetting problem is defined as a phenomenon where the performance of the old tasks dramatically decreases with the presence of new tasks. To further achieve better performance in a complex data stream environment, a continual learning algorithm should also be able to handle two major challenges, i.e., concept changes and label availability. The former problem causes a model trained using previously seen data batch being out-of-date. The latter challenge indicates that the labeling process should be accomplished and is subject to access to the ground truth. Further, there may exist a semi-supervised or unsupervised situation where the labels are very less or never available, respectively. These challenges demand an algorithm that is capable of adapting to concept changes with or without labels. Separately, a deep neural network (DNN) has shown promising results in processing any form of data. However, the static and offline nature of DNN hinders its implementation to address the evolving characteristics of data streams. The goal of this research is to present step-by-step solutions in circumventing the data stream problems using a flexible DNN which is able to incrementally increase or decrease its capacity in respect to problem complexity. The first contribution is presented in the third chapter. In this chapter, the problem of how to incrementally increase the network capacity in respect to problem complexity is handled. An incremental learning algorithm of DNNs for evolving data streams is proposed, namely Autonomous Deep Learning (ADL). It characterizes a flexible network structure where its hidden nodes and hidden layers can be incrementally constructed from scratch. The network significance (NS) formula derived from bias-variance trade-off The bias-variance tradeoff states that as model complexity rises, bias decreases but variance increases, resulting in a U-shaped test error curve is utilized to control the network complexity. This formula works by monitoring the possible underfitting and overfitting situation of a DNN. To further identify the loss of generalization power, a Drift Detection Scenario (DDS) is carried out to signal a real drift situation. This triggers the construction of a new hidden layer. The concept of different depth network structure is put forward to specifically combat the catastrophic forgetting problem because of hidden layer evolution. This network structure enables ADL to retain the constructed knowledge in the previous layers and thus maintaining the predictive performance whenever a new layer is added. In the fourth chapter, we explore the strategy to expand the capacity of MLP structure in respect to the problem complexity. We formalize this method as a Neural Network with Dynamically Evolved Capacity (NADINE). The main bottleneck to evolve an MLP structure is the loss of performance whenever a new hidden layer is added. We called this problem "the catastrophic forgetting induced by structural evolution". The major cause of this is the random initialization of the newly added layer. This problem is mitigated by using a sample and replay mechanism. It is understood that retaining samples and retraining the model should be avoided in the context of data stream learning. Because of that reason, a method to collect important samples, namely adaptive memory, from the streaming data is developed. The collected samples are then used to retrain the network whenever there is a real drift confirmed by DDS. Because every hidden layer is constructed in different concepts, it is important to govern the amount of update in each hidden layer according to its relevance to the output. The soft-forgetting strategy is proposed to handle this objective where it translates the correlation of each hidden layer and output to independent learning rates for every hidden layer. In the fifth chapter, we revisit the concept of denoising autoencoder (DAE) as a generative training mechanism that addresses the random initialization problem of DNNs. From the viewpoint of continual learning in the streaming environment, it encourages DNNs to adapt to concept changes while waiting for the labeling process. Being motivated by these facts, we proposed a deep evolving denoising autoencoder (DEVDAN). The main feature of DEVDAN lies in the coupled-generative-discriminative-learning phase which enables DEVDAN to exploit both labeled and unlabeled samples. The coupled learning phase is executed alternately whenever a data batch comes creating a continual learning cycle. The evolving trait of DEVDAN is carried out in both phases attempting to handle concept changes with or without the labels. In the generative phase, the evolving mechanism is controlled by the NS formula derived from the squared reconstruction error. The general trend of deep learning research changed dramatically during the midst of the work on this thesis with the presentation from Yann LeCun in which he stated that unsupervised and semi-supervised learning problems are the major challenges of AI for the next decade. Semi-supervised learning behaves similarly to supervised learning in which it learns a function from input-output pairs of data. However, they have additional features to address the lack of labeled samples. That is, the algorithms can automatically generate labels from the input data. In the viewpoint of evolving data streams, this can be adopted to circumvent semi-supervised or infinitely delayed labels situation in data stream environments. The sixth chapter in this thesis presents a self-evolving deep neural network, namely Parsimonious Network (ParsNet), as a solution to the semi-supervised data stream problem. It incorporates the coupled-generative-discriminative-learning phase exploiting both labeled and unlabeled samples. Further, a self-labeling strategy with hedge (SLASH) completes the coupled-learning phase by augmenting labeled samples and minimizing the effect of self-labeling mistakes in a semi-supervised situation. A deep convolutional network (CNN) is desired for data streams since it is able to extract better features especially in the case of unstructured data streams. Nonetheless, CNN training for data stream is hindered by its static nature and the expensive labeling cost. These demands call for an unsupervised data stream algorithm that is capable of processing both structured and unstructured data. In the seventh chapter, an Autonomous Deep Convolutional Network (ADCN) is presented. It combines the generative learning phase and self-clustering to learn better representation from the data in an unsupervised manner. The generative network involves a flexible MLP structure that is capable of evolving the hidden nodes and hidden layers on demand guided by NS formula and DDS, respectively. The self-clustering mechanism is performed in the deep embedding space of every layer while the final output is inferred by summing up the local output. Also, a latent-based regularization is employed to enable ADCN in circumventing the catastrophic forgetting problem in a continual learning environment. From the numerical study of NADINE, it is found that model selection plays a key role in the successful application of CNN. This triggers the development of an autonomous method that can generate an appropriate structure for a given problem. Most of the research pursues two directions, that is Reinforcement Learning (RL-based) and Evolutionary Computation based (EC-based) approaches. Those approaches, however, require huge computational resources since all of them generate architecture in a trial-and-error manner. Because of this reason, a novel data-driven architecture learning method, namely the Autonomous CNN (AutoCNN), is proposed in the eighth chapter. It is capable of incrementally constructing a CNN architecture for a given problem from scratch. It differs itself from EC and RL-based methods since it is a data-driven which does not involve any trial-and-error mechanism. The CNN layer growing, filter pruning and early stopping condition are combined to realize the structural evolution. A novel Feature Separability Score (FSS) is proposed to control layer growing which is able to measure the feature separability of the CNN network. A high FSS value triggers the addition of convolutional layers attempting to obtain more meaningful representation. The numerical studies show that the NS, DDS, and FSS formulas are successful in guiding the structural evolution on given problems exploiting both structured and unstructured data. This is achieved with a reasonable execution time which obeys the limitation given in data stream environments. The different-depth network structure and adaptive memory mechanism appear to be the key components in mitigating "the catastrophic forgetting induced by structural evolution". Separately, the coupled-generative-discriminative-learning phase and SLASH strongly form a holistic solution to execute semi-supervised learning in a non-stationary environment. Another important finding is that the combination of self-clustering mechanism and latent-based regularization delivers better accuracy compared to most baselines in dealing with unsupervised learning problems. Further, this mechanism successfully reduces the risk of catastrophic forgetting in continual learning environments. That is, being able to deliver a good performance on the previous task after learning a new task. These findings support the implementation of a flexible DNN and the proposed learning policies as a continual learning algorithm in a complex data stream environment. The proposed methods are applicable and, in some cases, have been applied to other kinds of tasks, i.e., transfer learning and control algorithm, but this thesis does not explore such application. The sources codes and raw numerical results of this thesis are available in this link https://tinyurl.com/AutonomousDL. To conclude, this thesis presents six major contributions to the study on data stream learning using flexible DNNs as follows: Development of a flexible different-depth DNN for data stream learning which is proficient in addressing concept drift and catastrophic forgetting due to structural evolution. Development of an MLP structure with dynamic model capacity for data stream learning without catastrophic forgetting. Development a flexible DNN with a coupled-generative-discriminative learning phase for exploiting both labeled and unlabeled samples in streaming environments. Development of a flexible DNN for handling the lack of labeled samples situations in streaming environments. Development of an evolving DNN for unsupervised and continual learning in streaming environments which can prevent catastrophic forgetting. Development of a learning mechanism to incrementally construct a CNN structure for a given problem.Doctor of Philosoph

ANALISIS HASIL PERANCANGAN PEMBANGKIT LISTRIK TENAGA SEPEDA STATIS DI FITNESS CENTER TERMINAL TRANSIT BAHAN BAKAR MINYAK PERTAMINA WAYAME AMBON

Nowadays, energy efficiency is an issue that is being faced by the industry. By increasing energy efficiency, production costs can be reduced to its lowest level. Terminal Transit Bahan Bakar Minyak Pertamina Wayame, Ambon as a part of the government committed to get actively involved in the implementation of the energy efficiency program. The innovation is done by converting the mechanical energy of the stationary bike to electrical energy by the generator mounted on the bike. Workers who do exercise by nature are wasting some energy. This energy of a worker is converted to mechanical energy by the stationary bike. Mechanical energy of the stationary bike is converted to electrical energy by DC generator mounted on the bike, which is the ultimate output. Finally, the electrical energy generated by stationary bike will then be stored in the battery. The energy stored in the battery is ready to be used as an energy source for LED lights scattered on the building. The results show that the stationary bike generator can generate 0.12 kW of power and save Rp. 468,594.30 per year

Online semisupervised learning approach for quality monitoring of complex manufacturing process

Data-driven quality monitoring is highly demanded in practice since it enables relieving manual quality inspection of the product quality. Conventional data-driven quality monitoring is constrained by its offline characteristic thus being unable to handle streaming nature of sensory data and nonstationary environments of machine operations. Recently, there have been pioneering works of online quality monitoring taking advantage of online learning concepts in the literature, but it is still far from realization of minimum operator intervention in the quality monitoring because it calls for full supervision in labelling data samples. This paper proposes Parsimonious Network++ (ParsNet++) as an online semisupervised learning approach being able to handle extreme label scarcity in the quality monitoring task. That is, it is capable of coping with varieties of semisupervised learning conditions including random access of ground truth and infinitely delayed access of ground truth. ParsNet++ features the one-pass learning approach to deal with streaming data while characterizing elastic structure to overcome rapidly changing data distributions. That is, it is capable of initiating its learning structure from scratch with the absence of a predefined network structure where its hidden nodes can be added and discarded on the fly in respect to drifting data distributions. Furthermore, it is equipped by a feature extraction layer in terms of 1D convolutional layer extracting natural features of multivariate time-series data samples of sensors and coping well with the many-to-one label relationship, a common problem of practical quality monitoring. Rigorous numerical evaluation has been carried out using the injection molding machine and the industrial transfer molding machine from our own projects. ParsNet++ delivers highly competitive performance even compared to fully supervised competitors.National Research Foundation (NRF)Published versionThis project was financially supported by National Research Foundation, Republic of Singapore, under IAFPP in the AME domain (contract no. A19C1A0018)

The effects of energy storages on small signal stability of a power system

Utilizing additional devices for small signal stability enhancement of power systems is crucial over the last few years due to increasing number of demand, uncertainty of load and integrating intermittent power system based on renewable energy. Among the number of additional devices in power system, energy storage is becoming more popular in last few decade. This paper investigates the impact of different types of energy storage on the small signal stability of power system. In order to analyze the impact of aforementioned energy storage for small signal stability, a single power system connected to an infinite bus is used as a test system. The eigenvalue and damping ratio of the investigated mode are used to examine the impact of energy storage on dynamic behavior of the investigated system. Furthermore, a time domain simulation is used to validate the eigenvalue analysis of the system. From the simulation, it is found that the type of energy storage have significant influence on the small signal stability of power system, indicated by the increasing value of damping over the system